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(57) ABSTRACT 

Methods and systems that classify sepia-toned digital 
images are described. An exemplary method includes con- 
verting RGB values of a digital image to hue saturation 
intensity (HSV) values. Pixels of the digital image that are 
not above a predetermined intensity or saturation threshold 
are removed from the image. The remaining pixels are 
analyzed and classified as sepia-toned images based on the 
results of the analysis. Additionally, a method for training a 
system to recognize sepia-toned images is disclosed. 

20 Claims, 3 Drawing Sheets 
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1 2 

CLASSIFICATION OF PHOTOS WITH SEPIA- ^techniques include the following, articles ^S.F* Chang„W;r 

TONES Chen and H. Sundaram, "Semantic visual templates: linking 

visual features to semantics," Proc. of IEEE Intl. Conf. on 
Image Processing, vol. 3, p. 531-35, 1998. S. Paek and S. -F. 

BACKGROUND OF THE INVENTION 5 Chang, "A knowledge engineering approach for image clas- 

* r- u c i_ t • sification based on probabilistic reasoning systems/' Proc. 

1. Field of the Invention ^ Jml Qonf Qn Muhimedia & Exp0y 2 ooo. R. Qian, N. 

The present invention relates to digital image database Haering and I. Sezao, "A computational approach to seman- 

and classification systems, and in particular to classification event detection," Proc. of IEEE Conf. Computer Vision 

of sepia tone photographs that have been converted to digital 10 and Pattern Recognition, vol. 1, p.200-06, 1999. M. Szum- 

formats. mer anc j r ^ Picard, "Indoor-outdoor image 

2. Background Information classification," Proc. of IEEE Workshop on Content-Based 
Automatic image classification has many important Access of Image and Video Libraries, p.42-51, 1998. A. 

applications, particularly in classifying a special class of Vailaya, M. Figueiredo, A. Jain and H.-J. Zhang, "A Baye- 
images:, black and white photos with sepia tones. Some of 15 sian framework for semantic classification of outdoor vaca- 
these images have been scanned from antique photos, which tion images," Proc. of SPIE: Storage and Retrieval for 
were originally in black and white. Over the years they Image and Video databases VII, vol 3656, p. 415-26, 1999. 
gradually turn to yellow or brown due to the chemical A. Vailaya and A. Jain, "Detecting sky and vegetation in 
reactions of the film paper. Still, many other sepia-tone outdoor images," Proc. of SPIE: Storage and Retrieval for 
images were specially produced either chemically or digi- 20 Image and Video databases 2000, vol. 3972, p. 411-20, 
tally to generate an antique appearance. 2000. N. Vasconcelos and A. Lippman, "A Bayesian frame- 
Classification of photos with sepia tones is useful in wo * semantic content characterization/' Proc. of IEEE 
several applications. Because of the special color tones, Con f' on Computer Vision and Pattern Recognition, p. 
many general-purpose color image processing techniques 566-71, 1999. 

would not work well on these images. For example, a 25 Image databases frequently contain images having differ- 
face-detection algorithm in an imaging system that detects ent characteristics, such as color images of different color 
faces purely based on the skin color is unlikely to generate resolutions (e.g., 16, 256, 16 bit, and 24 bit color), gray scale 
the correct results on these photos because the skin color has images, black and white images, sepia-toned images, and the 
been changed. By detecting images with sepia tones, the ^ 0T image classification techniques do not address 
system can handle these images appropriately to avoid 30 automatic sepia-toned image classification. Since sepia- 
possible errors. For instance, an alternative algorithm can be toned images can cause difficulties when used with conven- 
used to detect the face that uses features that are invariant to tional image processing techniques and many sepia tone 
color changes, such as eye corners. Therefore, the detection images are valuable antique photos, it would be desirable to 
accuracy of the image recognition system may be signifi- have aQ image classification system that can analyze the 
cantly improved. 35 properties of the images themselves and classify the images 
Sepia-toned images can also cause problems in automatic ?<»«•*« l ° whether ° r n ° l "*age is a sepia-toned 
color balance or enhancement processes. Because the colors image. 

are narrowly concentrated in the sepia tones, the images SUMMARY OF THE INVENTION 

could be mistakenly considered as color imbalance^ Color ^ ^ t iavention fe to methods ^ ^ 

corrections to these images could cause undesirabk cobr that classify digital images containing sepia-tones. An exem- 

artifacts. If a color imaging process knows beforehand what { method ^ coavertirjg values of a f^t color 

kind of image is being processed, it can react appropriately ^ of a image to hue 

and achieve better results. values; removing any pixel of the digital image below at 

Another application of sepia-tone image classification is 45 least one of a saturation threshold and an intensity threshold; 

in image indexing. Urge image databases or collections analyzmg) aft er the step of removing, remaining pixels of 

require good indexing mechanisms so that images can be me image; and classifying tne digital image as a 

categorized effectively, browsed efficiently, and retrieved sepia-toned image based on the analysis of the remaining 

quickly. Conventional systems store and retrieve specific pixels of the digital image 

information from the database using, for example, descrip- 50 Alternatc cmbodiments providc for estimaling me prob . 

tive information regarding the image file, such as file ere- LT( t , Ct , . • • T • • * ju j 

, . ,., ™ • <■ ability that each ot the remaining pixels is sepia-toned based 

ation date, file name, file extension and the like. This form n ' A . • . j- f -J,,!- rtf f , • 

„. . . , , on a predetermined color distribution ot sepia-toned images 

of unage classification is not significantly different from the , j , • . . . , • 

.° . e , and determining a probability that the digital image is a 

classification of any other digital information. • . j - 

3 & sepia-toned unage. 

By relying on the file information only cursory informa- 55 ^ cxelrj plary method of training a system to detect 

tion can be obtained about the file and nothmg at all ia ., oned converting values of a first 

specifically related to the image For example, an image file ^ of a luraH of ^ iffl to HSV yalues> 

could have a name that had no relation to the type or content wherein me trainin j m images - remov . 

of the image, such as a black and white image could have the m ^ Qf t(je ^ ^low at least one of a 

file name color image . Other systems provide classmca- 60 saturation (hreshold and an mtensit threshold for each of 

Uon based on the content of the images, such as flowers, ffie , uraU of im and ^ the ste p 

dogs, and the like. In practice this is usually done by of rcmovm me ^ distribution of H SV values for 

keyword annotation, which is a laborious task. remaining pixels over the plurality of training images. 

Image classification techniques have been proposed in the 

past years that are designed for use in image databases. 65 BRIEF DESCRIPTION OF THE DRAWINGS 

However, none of these prior techniques have addressed The above features and advantages of the invention, and 

identifying sepia-toned images. Examples of these prior additional features and advantages of the invention, will be 
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i better appreciated; from, the following detailed description, of 
the invention made with reference to the drawings, wherein: 

FIG. 1 shows a flow chart of an exemplary method of the 
present invention; 

FIG. 2 shows a flow chart of a training method of the 
present invention; 

FIG. 3 shows a block diagram of an exemplary system of 
the present invention; and 

FIG. 4 shows a block diagram of an exemplary training 
system of the present invention. 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENTS 

FIG. 1 shows a flow chart of an exemplary method of 
classifying sepia-toned images. In step 100, values of a first 
color space of a digital image are converted to hue saturation 
intensity values (HSV). One skilled in the art will appreciate 
that the digital image can be retrieved from a plurality of 
digital images stored in an image database containing at 
least one of a group consisting of color images, gray scale 
images and sepia tone images. Additionally, the first color 
space can be any know representation of the color space. 
However, for purposes of explanation and not limitation the 
remaining description will refer to the first color space as 
Red, Green, Blue (RGB) color space. Therefore the conver- 
sion described is RGB to HSV. 

In an exemplary method, the RGB values are normalized 
(i.e., ranging from 0-1.0). The conversion formula can then 
be given as: 

Max = max(/f, C, B) 
Min = min(/?, G t B) 
Val = Max 

If (Vo/ = 0) Sal=0, else Sal = I -Min /Val 

If (Sat = 0) Hue is undefined, else 

(C-fl)/(Max-Min)x60 if {R = Max A(G-5) * 0) 
{G-B)f (Max - Minj X60+-360 if (R = Max A (G - B) < 0) 
(2.0 + (B - R)l (Max - Min)) x 60 if (G = Max) 
(4.0 + (R - C)/ (Max - Min)) x 60 if (B = Max) 

where the range of H is 0-360 degrees, S is 0-1.0, and V is 
O-1.0. However, those skilled in the art will appreciate that 
other conversion formulas could be used. Additionally, it 
will be appreciated that black and white images can be 
directly identified at the RGB to HSV color space conver- 
sion step and can be removed from further consideration. 

In step 110, any pixel of the digital image below at least 
one of a saturation threshold and an intensity threshold is 
removed. Pixels having low saturation or low intensity are 
removed from consideration because they tend to appear as 
gray regardless of their hue values. The pixels are removed 
based on the following criteria: 

(l-5) 3 +(l-V) 3 -^ 

where c is a threshold set 0.94, S is the saturation value 
ranging from 0 to 1.0 and V is the intensity value ranging 
from 0 to 1.0. 

Id step 120, after the step of removing, the remaining 
pixels of the digital image are analyzed. The digital image is 
classified as a sepia-toned image based on the analysis of the 
remaining pixels of the digital image, in step 130. The 
classification includes at least one of retrieving, displaying, 



^ tagging, storingi.and -the -like,. thatcan ho performed! on.the 
image. For example, the process could tag the sepia-toned 
images in an image database so another program could 
preform special image processing procedures on the tagged 

5 sepia-toned images or retrieve images from the image data- 
base based on the sepia-tone tag. 

Optionally, in step 122, the analysis is performed by 
determining a probability that each of the remaining pixels 
is sepia-toned based on a predetermined color distribution of 

10 sepia-toned images. The predetermined color distribution 
can be determined in a training stage that is discussed later. 
Then, in step 124, a probability that the digital image is a 
sepia-toned image is determined. In step 126, the probability 
that the digital image is a sepia -toned image is compared to 

15 a predetermined threshold for identifying sepia-toned 
images. For example, if the probability that the digital image 
is a sepia-toned image is greater than the predetermined 
threshold the image is determined to be sepia-toned. Those 
skilled in the art will appreciate that the predetermined 

20 threshold for identifying sepia-toned images can also be 
determined during a training stage. 

Referring to FIG. 2, a flow chart of a method of training 
a system to detect sepia-toned images is shown. In step 200, 
the RGB values of a plurality of training images are con- 

25 verted to HSV values. The training images are preferably 
sepia-toned images. Those skilled in the art will appreciate 
that images that are not sepia-toned images could be used in 
the set of training images. However, the training images 
would then have to be identified as sepia-toned or not. In 

30 step 210, any pixel of the training images below at least one 
of a saturation threshold and an intensity threshold is 
removed for each of the plurality of training images. The 
color distribution of HSV values for the remaining pixels is 
then estimated over the plurality of training images, in step 

35 230. Optionally, in step 240, a threshold is established at 
which an image is classified as a sepia-toned image. Those 
skilled in the art will appreciate that the RGB to HSV 
conversion, the saturation threshold and the intensity thresh- 
old for the training process can be determined as described 

40 above for the classification process. Therefore, the descrip- 
tion and equations' will not be repeated. 

To facilitate an understanding of the invention, many 
aspects of the invention are described in terms of sequences 
of actions to be performed by elements of a computer-based 

45 system. It will be recognized that in each of the 
embodiments, the various actions can be performed by 
specialized circuits (e.g., discrete logic gates interconnected 
to perform a specialized function), by program instructions 
being executed by one or more processors, or by a combi- 

50 nation of both. Moreover, the invention can additionally be 
considered to be embodied entirely within any form of a 
computer readable storage medium having stored therein an 
appropriate set of computer instructions that would cause a 
processor to carry out the techniques described herein. Thus, 

55 die various aspects of the invention can be embodied in 
many different forms, and all such forms are contemplated 
to be within the scope of the invention. For each of the 
various aspects of the invention, any such form of an 
embodiment is referred to herein as "logic that" performs a 

ea described action. 

FIG. 3 is a block diagram of a computer-based system 300 
using the threshold obtained from the training process 
described above. For example, as previously described, each 
new input image 310 goes through the RGB to HSV color 

65 space conversion using logic that converts values of a first 
color space of a digital image to hue saturation intensity 
(HSV) values, in block 320. The image's low saturation or 
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iintensityvpixeU; are,, them -removed; using4ogic4haUremoye& ; . ^What is claimed 

any pixel of the digital image below at least one of a 1. A method of classifying digital images comprising: 

saturation threshold and an intensity threshold, in block 340. „„i,.„ n c „ „«™ ~f „ -^i • 

„ . . . . j . v . ' . f convertmg values or a first color space of a digital image 

The remaining pixels are used to estimate the probability of , u *. *■ • * / T ,n, A i 

.... i . . „ * * ii iu j to hue saturation intensity (HSV) values; 

{ this unage being sepia-toned using a maxunum likelihood 5 J v y 

J estimation. Each pixel is assumed to be an independent removing any pixel of the digital image below at least one 

^sample of the input data. The probability of pixel i being of a saturation threshold and an intensity threshold; 

sepia-toned can be estimated directly from the predeter- analyzing, after the step of removing, remaining pixels of 

mined color distribution provided by block 350 (e.g., as the digital image; and 

determined from the mining process from FIG 2). Then, a classiiying the digiud image as a sepia-toned image based 

joint log probability log (P can be calculated for the entire O0 the ^ of me * emainin £ ^ of th * ^ 

unage, using logic that analyzes the remaining pixels of the 3 & r ^ 

digital image, in block 360, by: - if? 086 ' 



2. The method of claim 1, wherein analyzing comprises: 
determining a probability that each of the remaining 
15 pixels is sepia-toned based on a predetermined color 

logP) = ^—^ — distribution of sepia-tooed images; and 

determining a probability that the digital image is a 

where N is the total number of remaining pixels, p ; is the sepia-toned image. . .. , .... . i . 

probability of pixel i being sepia-toned, and log(P) indicates 20 A . 3 ; ™ e method of claim 2 > ^ herem * c P'obabihty that the 

the probability of the image being sepia-toned. The calcu- digital image is a sepia-toned image is determined by the 

lated probability values of sepia-toned photos are, in following: 
general, much higher than normal color photos. Therefore, a 

simple threshold supplied by block 370 can be used to ^ lo 8tP*) 

separate color images from sepia-toned images. For . _ _» 

example, the threshold is set to -5.2. Thus, if log(P) is 25 osl } ~ n 
greater than -5.2, the image is determined to be sepia-toned, 

using logic that classifies the digital image as a sepia-toned , , , „ „ 

image based on the analysis of the remaining pixels of the therein n is a total number of the remaining pixels, p, is the 
digital image, in block 380. probability of each remaining pixel i being sepia-toned, and 
FIG. 4 is a block diagram of a computer-based training 30 lo g( p ) indicates the probability of the image being sepia- 
system 400 according to another embodiment of the present toned. 

invention. For example, as previously described, each train- 4. The method of claim 3, further comprising: 

tog image 410 goes through the RGB to HSV color space lhe probability that the digital image is a 

conversion in block 420. The tow saturate , or intensity sepia-toned image to a predetermined threshold for 

pixels are then removed in block 440 In block 460 the HSV 35 ideDtif y ing sepia-toned images. 

space is divided uniformly into cells along H S and V 5 ^ method of claim ^ wherein , he predetermined 

dimensions. For example the H dimension can be divided mreshold for ; denti fy mg sepia-toned images is -5.2, and 

into thirty-six cells, the S dimension into four cells, and the wherein the ^ fe dassified ^ a ia _ toDed ^ 

V dimension into four cells. Each of the remaining pixels of ^ loefP) greater than -5 2 

the training images is then assigned to one of the cells based 40 fi ^ method of c , aim { the di itaJ ^ ^ OM 

on its HSV values. Values for each cell are established by ta of an ; database containing at least one of a 

counting the remaining pixels in each celL The values for consisting of color images, gray scale images and 

each cell are then normalized by dividing the value for each _ • . , • ^ 

. . , , <. * . . . • ™» t sepia-tonea images, 

cell by the total number of the remaining pixels. The result 7 ^ fflethod of daim j wherein cUssi£ying comprises: 

is a thirty-six by four by four-dimensional vector that 45 . 7 

characterizes the color distribution of the training images, in performing at least one of retrieving displaying, tagging, 

block 480. This color distribution can also be seen as a rough „ ™? storing the unage based on the step of classifying, 

estimation of the probability density function for remaining 8 ' ^ m ,f th ° d ° f cla , m \ Wbereln *° 006 ° f a 

pixels in the training images. TypicaUy, the colors in sepia- M f ur ^ n . threshold mt6QSlly threshoId ' * de,er - 

toned images are mostly centered around yellow and red. 50 mmed DV - 

Therefore, the cells that correspond to these colors will (i-5) 3 +(i-v) 3 ^c 3 
contain the majority of the remaining pixels. 

Hie foregoing has described principles, preferred embodi- wherein c fa the threshold> s is the saturation value and V is 

ments and modes of operation of the invention. However, ^ e intensity value 

the invention is not limited to the particular embodiments 55 „ ™ . j " r t • o L - u * n c 

. . „ . « • « . . . 9. The method of claim 8, wherein c is about 0.94, S 

discussed above. For example, each sepia-toned image could - rw^jTr c r»*^^ 

- 4 . , , . a , . K « , . u-i% t *u * ranges from 0 to 1.0 and V ranges from 0 to 1.0. 

further be classified based on its probability. Images that & a 

were very close to the threshold value could then be tagged 10 - The method of claim 1, wherein the first color space 

for manual review. Additionally, if the majority of sepia- ^ RGB aod wherein the RGB values of the digital image are 

toned images were near the threshold, this could serve as 60 normalized to range from 0 to 1.0. 

indication that a new set of training images is needed to 11 ■ The method of claim 10, wherein the conversion from 

retrain the system. RGB to HSV, is determined by: 

Therefore, the above -described embodiments should be 

regarded as illustrative rather than restrictive, and it should Max = maxfrt, c, B) 

be appreciated that variations may be made in those embodi- 65 

ments by those skilled in the art, without departing from the ^ " rmi * /f * G * fl) 
scope of the invention as defined by the following claims. 
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Hue = 



7 

j . ■■ ^"V^-continueoV- 1 ■ : ws*--.'* ••;« ^ 

Val = Max 

If [Val-0) Sat = 0, else Sat = 1 - Min /Val 

If = 0) Hue is undefined else 

f(G-5)/(Max-Min)x60 if (R - Max A (G - 5) £ 0) 
| (G-fl)/(Max-Min)x60 + 360 if (/? = Max A(G-5) < 0) 
(2.0 + (B - /?)/(Max - Min)) X 60 if (G = Max) ' 
[ (4.0 + {R- C)/(Max - Min)) x60 if {B = Max) 



8 



10 



wherein the range of H is 0-360 degrees, S is 0-1.0, and V 
is 0-1.0. 

12. A method of training a system to detect sepia-toned 
images comprising: 

converting values of a first color space of a plurality of 
training images to HSV values, wherein the training 
images are sepia-toned images; 

removing any pixel of the training images below at least 
one of a saturation threshold and an intensity threshold 
for each of the plurality of training images; and 

estimating, after the step of removing, the color distribu- 
tion of HSV values for remaining pixels over the 
plurality of training images. 

13. The method of claim 12, further comprising: 
establishing a threshold at which an image is classified as 

a sepia-toned image. 

14. The method of claim 12, further comprising: 
dividing HSV space uniformly into cells along H, S, and 

V dimensions; 

assigning each of the remaining pixels of the training 
images to one of the cells based on the HSV values of 
each remaining pixel; 

establishing values for each cell by counting the remain- 
ing pixels in each cell; and 

normalizing the values for each cell by dividing by a total 
number of the remaining pixels. 

15. The method of claim 14, wherein there are thirty six 
cells in the H dimension, four cells in the S dimension and 
four cells in the V dimension. 

16. The method of claim 12, wherein the least one of a 
saturation threshold and an intensity threshold, is deter- 
mined by: 



20 



25 



wherein c is the -threshold^ Si is ithe saturation value.and V?is- 
the intensity value. 

17. The method of claim 16, wherein c is about 0.94, S 
ranges from 0 to 1.0 and V ranges from 0 to 1.0. 

18. The method of claim 12, wherein the first color space 
is RGB and wherein the RGB values of each training image 
are normalized to range from 0 to 1.0. 

19. The method of claim 18, wherein the conversion from 
RGB to HSV, is determined by: 

Max = max(/?, G, B) 
Min = mil*/?, C, B) 
Val = Max 

If {Val = 0) Sai = 0. else Sat = I - Min/ Val 
If {Sat = 0) Hue is undefined, else 



Hue = 



(G-fl)/(Max-Min)x60 if (/? = Max A(G- B) 0) 
{G-B)f (Max - Min) x 60 + 360 if (R = Max A (G - B) < 0) 
(2.0 + (B - /?)/(Max - Min)) x 60 if (G = Max) 
(4.0 + (R - G)/(Max - Min)) x 60 if (B = Max) 
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wherein the range of H is 0-360 degrees, S is 0-1.0, and V 
is 0-1.0. 

20. A computer-based system for classifying digital 
images comprising: 

logic that converts values of a first color space of a digital 
image to hue saturation intensity (HSV) values; 

logic that removes any pixel of the digital image below at 
least one of a saturation threshold and an intensity 
threshold; 

logic that analyzes, after the step of removing, remaining 
pixels of the digital image; and 

logic that analyzes classifies the digital image as a sepia- 
toned image based on the analysis of the remaining 
pixels of the digital image. 
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